Prosodic Cues in Multimodal Speech Perception

Potential visual prosodic cues for prominence and phrasing comprising eyebrow movements were manipulated using a system for audio-visual text-to-speech synthesis which has been implemented based on the KTH rule-based synthesis. Two functions of prosody (prominence and phrasing) were tested in two separate experiments. A test sentence, ambiguous in terms of an internal phrase boundary, was used for both experiments. Acoustic cues and lower face visual cues were held constant for all stimuli. Upper face cues were eyebrow movement where the eyebrows were raised on successive words in the sentence. The results indicate a general coupling between eyebrow raising and perceived prominence while suggesting a more complicated relationship between eyebrow movement and phrasing. The results also point against a tight perceptual connection between F0 and eyebrow movement indicating that selective eyebrow movement can be effective as an independent prosodic cue to prominence.